feat: implement Firestore event and error storage (Issue #2)#12
Merged
feat: implement Firestore event and error storage (Issue #2)#12
Conversation
- Add tenacity dependency for retry logic
- Create FirestoreEventStore class with Firestore client init
- Implement _get_doc_ref() for events/{stream}/items/{id} path construction
- Implement _event_to_dict() for datetime serialization
- Add stream field to TypedEvent for routing support
- Add comprehensive unit tests for setup and helpers
- All tests passing with 100% coverage on new code
- Implement store() for single event writes with asyncio.to_thread wrapper - Implement store_batch() with automatic 500-event chunking - Add _write_with_retry() and _write_batch_with_retry() with tenacity - Exponential backoff on DeadlineExceeded, ServiceUnavailable, InternalServerError - Add comprehensive unit tests for write operations and retry behavior - Test batch chunking (501 events → 2 batches) - Test retry success after 2 failures - Test retry exhaustion raises RetryError after 3 attempts - All tests passing with 100% coverage on new code
- Implement read(stream, limit) for querying events by stream - Query returns events in reverse chronological order (newest first) - Add _dict_to_event() for deserializing Firestore docs to TypedEvent - Handle IdentifyEvent, TrackEvent, PageEvent deserialization - Convert ISO timestamp strings back to datetime objects - Add retry logic to read operations - Add comprehensive unit tests for read operations - Test empty stream returns empty list - Test deserialization for all event types - Test ValueError on unknown event_type - All tests passing with 100% coverage on new code
- Create FirestoreErrorStore class for failed event storage - Implement store_error() for capturing payload, error, timestamp, metadata - Implement query_errors() for retrieving errors in reverse chronological order - Use flat errors/ collection structure for simple querying - Add retry logic to error storage operations - Add comprehensive unit tests for error store - Test error storage with and without metadata - Test error querying and empty results - All 20 unit tests passing with 100% coverage on new code
- Add docker-compose.yml with Firestore emulator - Add 3 integration tests: - test_write_and_read_events: Write 100 events, read back, verify - test_stream_isolation: Verify mobile/web streams are isolated - test_error_store_persistence: Verify DLQ stores and retrieves errors - Add integration marker to pytest.ini - Tests skip gracefully when FIRESTORE_EMULATOR_HOST not set - Update README with Docker Compose usage instructions - Add asyncio_mode to pytest.ini for async test support - All integration tests pass when emulator is running
- Update specs/core-pipeline/tasks.md acceptance criteria - All 7 acceptance criteria for Task 4 now marked complete [x] - FirestoreEventStore and FirestoreErrorStore fully implemented - 20 unit tests passing with 100% coverage - 3 integration tests passing with Docker Compose - Comprehensive docstrings on all public methods
prosdev
force-pushed
the
feat/firestore-storage
branch
from
de58d55 to
17cafd1
Compare
- Add return type annotations to _read_with_retry() and _query_errors_with_retry() - Use Any for Firestore generator return types - Add mypy to pre-commit hooks - All type checks now pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements persistent storage for events and errors using Google Cloud Firestore with batch operations, retry logic, and stream-based isolation.
Closes #2
Implementation Details
FirestoreEventStore
asyncio.to_thread()to wrap sync Firestore clientstore(),store_batch(),read(stream, limit)FirestoreErrorStore (Dead Letter Queue)
errors/collectionstore_error(),query_errors(limit)Testing
Docker Compose
docker-compose.ymlwith Firestore emulatorCommits
store(),store_batch()with tenacity retriesread(),_dict_to_event()with type routingTesting
Files Changed
src/eventkit/stores/firestore.py(new, 240 lines)tests/unit/stores/test_firestore.py(new, 270 lines)tests/integration/test_firestore_integration.py(new, 140 lines)docker-compose.yml(new)pytest.ini(updated with integration marker)README.md(updated with Docker instructions)pyproject.toml(added tenacity dependency)src/eventkit/schema/events.py(added stream field)specs/core-pipeline/tasks.md(marked Task 4 complete)Dependencies Added
tenacity>=8.2.0- Retry logic with exponential backoffAcceptance Criteria (All Met)
{stream}/{timestamp}_{uuid}